Ogden  Air  Logistics  Center 


A  Quality  Process 
Performance  Model 
for  Software 
Development  Projects 

Using  Monte  Carlo  Simulation  to  Predict 
Interim  and  Final  Product  Quality 

David  R.  Webb 

Senior  Technical  Program  Manager 
Hill  Air  Force  Base,  Utah 

SSTC  2009 


SSTC  2009 


BE  AMERICA ’S  BEST 


1 


1ST  Process  Quality 
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■  Focus  on  Defects 

■  A  defect  is  defined  in  the  520th  Squadron  Quality 
Management  Plan  as 

“a  product  or  product  component  that  does  not  meet 
requirements  or  a  design  or  implementation  element  that  if 
not  fixed  could  cause  improper  design,  implementation,  test, 
use  or  maintenance” 

■  The  number  of  defects  in  the  product  is  only  one 
indication  of  product  quality 

■  Defects  cause  rework  and  become  increasingly 
expensive  to  fix 

■  Until  we  have  functional  software  with  relatively  few 
defects,  it  doesn’t  make  sense  to  focus  too  much  on 
the  other  quality  issues 


SSTC  2009 


BE  AMERICA ’S  BEST 


2 


A  Simple  Quality  Model 
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■  Our  processes  have  basically  2  kinds  of  defect- 
related  activities: 

■  Activities  when  defects  are  inadvertently  injected 

■  Activities  when  defects  are  sought  for  and  removed 
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Estimating  a  Project 
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■  Effort  and  Schedule 

■  Typically,  we  are  able  to  estimate  how  long  our 
schedule  will  take 

■  We  also  typically  break  those  estimates  down  into  the 
phases  our  process  -  this  becomes  our  WBS 
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Gathering  Historical  Data  -  1 
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■  Defect  Injection  Rate  (DIR) 

■  For  all  completed  projects,  we  should  examine  all  the 
defects  found  and  determine  during  which  phase  of 
our  process  they  were  introduced 

■  We  also  know,  once  the  project  is  complete,  how 
many  hours  were  spent  in  those  phases 

■  DIR  can  be  calculated  as  explained  below: 


dx  =  defects  injected  in  process  block  X 
aphx  =  actual  cost  performance  in  hours  for  block  X 
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Gathering  Historical  Data  -  2 
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■  Defect  Detection  Ratio  (DDR) 

■  As  with  DIR,  we  can  examine  closed  projects  to 
determine  during  which  phases  of  our  process 
defects  were  discovered 

■  We  also  know,  once  the  project  is  complete,  how 
many  total  defects  were  found  in  each  phase 

■  DDR  can  be  calculated  as  explained  below: 

DDRx = -tt~ 
ix+ex 

ix  =  all  defects  found  in  QA  activity  for  process  block  X 
ex  =  any  defects  injected  in  the  process  block(s)  covered  by  QA  activity  X  but 

detected  at  a  later  QA  activity 
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Completing  the  Quality  Model 
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■  Defects  Injected  (Dl) 

■  Now  that  we  know  the  DIR,  we  can  use  our  hours  estimate  to  project  how  many 
defects  will  be  inadvertently  injected  in  each  production  phase 

■  Defects  Removed  (DR) 

■  Also,  since  we  know  the  DDR  of  our  QA  phases,  we  can  project  how  many  of 
those  defects  will  probably  be  removed 

■  Defects  Remaining 

■  Determining  the  bugs  left  behind  is  easy:  Defects  Remaining  =  Dl  -  DR 
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Quality  Model  Issues 
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■  Effort  Estimation 

■  Productivity  isn't  always  what  you  estimate  it  will  be 
...  sometimes  you  use  more  hours  than  planned, 
sometimes  less. 

■  Quality  Estimates 

■  DIR  can  vary  based  upon  team  composition,  the 
product  being  produced,  the  familiarity  with  the 
product  and  tools,  etc. 

■  DDR  per  phase  varies  based  upon  the  same  kinds  of 
considerations. 

■  Updating  the  Model 

■  The  Model  must  take  into  account  the  variability  of 
effort,  defect  injection  and  defect  removal  to  be 
accurate 
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Accounting  for  Variability  in 
Effort 
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■  Effort  Estimating 

■  We  can  easily  calculate  a  project’s  Cost  Productivity 
Index  (CPI)  for  historical  projects 

■  CPI  is  the  ratio  of  planned  to  actual  hours  (or  dollars) 

■  We  can  divide  our  effort  estimates  by  CPI  to  get  a 
better  estimate  of  what  our  real  effort  will  be 

•A  project  that  consistently  overestimates  will  have  a  CPI  >  1; 
dividing  by  the  CPI  will  decrease  the  estimate 

•A  project  that  consistently  underestimates  will  have  a  CPI  <1; 
dividing  by  the  CPI  will  increase  the  estimate 

■  However,  just  as  CPI  is  not  the  same  for  every 
historical  project,  an  average  CPI  may  not  be 
sufficient  to  properly  adjust  our  effort  estimates 
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Accounting  for  Variability  Using 
Monte  Carol  Simulation 
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■  Monte  Carlo  Simulation 

■  a  technique  using  random  numbers  and  probability 
distributions  to  solve  problems 

■  Uses  “brute  force”  computational  power  to  overcome 
situations  where  solving  a  problem  analytically  would 
be  difficult 

■  Iteratively  applies  the  model  hundreds  or  thousands 
of  times  to  determine  an  expected  solution 

■  First  extensively  studied  during  the  Manhattan 
project,  where  it  was  used  to  model  neutron  behavior 
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How  Does  Monte  Carlo 
Work? 
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■  Monte  Carlo  Steps 

1.  Create  a  parametric  model 

2.  Generate  random  inputs 

3.  Evaluate  the  model  and  store  the  results 

4.  Repeat  steps  2  and  3  (x-1)  more  times 

5.  Analyze  the  results  of  the  x  runs 
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Applying  Monte  Carlo 
Simulation  to  the  Quality  Model 
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■  Variability 

■  Allow  the  following  values  to  be  variable 

•Cost  Productivity  Index 

•  Defect  Injection  Rate  per  Phase 

•  Defect  Detection  Ratio  per  Phase 

■  Use  Historical  Data  to  Determine 

•Statistical  distribution  of  data 
•Averages  and  limits  of  the  data 

■  Apply  Monte  Carlo 

■  Have  the  Monte  Carlo  tool  run  the  model  thousands  of 
times 

■  Each  time,  Monte  Carlo  will  choose  a  random  value  for 
CPI,  DIRs  and  DDRs,  generating  a  new  result 

■  Over  time,  a  profile  will  be  built  showing  the  distribution  of 
likely  outcomes 
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Historical  Variability 
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Setting  Up  the  Monte  Carlo 
Simulation 
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Quality  Model  for  Development  Projects 
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Some  Interim  Results 
(70%  Certainty) 
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Final  Results  of  the  Quality 
Model  (70%  Certainty) 
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Tracking  the  Project  Using  the 
Model 
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■  During  Planning 

■  Run  the  model 

■  Determine  projects  for  final  outcome  and  all  interim  outcomes 

■  Compare  the  final  outcome  to  project  goals 

■  If  goals  are  not  met,  then  a  process  improvement  is  warranted 
(i.e.,  changes  to  increase  DDR  or  decrease  DIR) 

■  During  Project  Execution 

■  Compare  the  interim  results  to  actual  results 

■  The  model  will  only  tell  you  the  MAXIMUM  number  you  should 
expect  within  your  certainty  level 

■  If  you  see  results  lower  than  that  number,  you’re  probably  OK 

■  If  you  see  results  much  higher  that  that  number,  then  you  need 
to  do  some  investigation 

■  Once  you  have  true  interim  results,  replace  the  Monte  Carlo 
variation  with  the  real  numbers  and  re-run  the  model  -  do  you 
still  have  a  final  outcome  that  meets  project  goals? 
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Examples  of  Tracking  the 
Project 
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as  actual  data  replace  the  Monte  Carlo 
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Model  Demonstration 
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Summary 
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■  The  Software  Maintenance  Group  at  Hill  Air  Force 
Base  has  created  a  Quality  Model  applicable  for 
most  software  development  projects 

■  Quality  is  modeled  by  predicting  defect  injection 
and  removal  using  historical  data 

■  Variation  is  taken  into  account  by  using  a  Monte 
Carlo  Simulation  to  adjust  estimates,  defect 
injection  rates  and  defect  detection  ratios 

■  Interim  results  can  be  used  to  guide  the  project 
toward  a  final  quality  goal 

■Actual  data  replaces  projected  in  the  model  as  the 
project  progresses 
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Questions 
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Contact  Information 
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